1. Field of the Invention
The present invention is directed to the field of motion detection and more particularly to the utilization of the detected motion to minimize its effect in the recording of an image.
2. Description of Related Art
It has been estimated that 15-20% of consumer pictures have "detectable image blur problems", and that camera shake and subject motion are significant contributors.
A relationship may be established between various shutter speeds and the maximum allowable motion at each speed such that an intelligent camera system always chooses a shutter speed such that motion of less than or equal to a "detectable" limit, in the finished print, is experienced while the shutter is open.
One approach is to hold the shutter open and determine how much movement will degrade the resultant picture "detectably" and "objectionably". Measurements of this motion have been made for typical cameras and for typical users. It has been determined that 0.001" of "equivalent linear smear" is detectable, and 0.002" is objectionable.
The possible motion of a camera has six degrees of freedom, including translational and rotational components. It has been determined experimentally however, that the rotational components of "pitch" and "yaw" dominate the motion at the film plane due to the magnification of the lens. Moreover, these components manifest themselves approximately as linear translations along the two orthogonal axes at the film plane. Therefore, "equivalent linear smear" presumes to take into account the actual motion, the camera's physical dimensions, and the lens focal length.
In the camera art, images captured from moving objects may be compared digitally in order to determine the amount and the direction of the camera's or the image's motion. Such determination is of particular utility in providing information that can be used to dampen the motion of the camera or to determine the motion of a subject. Once the images are captured, for example on film, they can be converted to a digital form and processed utilizing selected image processing methods. Image stabilization for hand-held cameras is important because it minimizes or eliminates the blurring of images captured from subjects that may be either still or moving. In any event, digital comparisons of the images formed in the camera's eye can be used to determine the amount and the direction of the images motion. Such detection is of particular utility in providing information not only for the handling of cameras but also for the determination of the movement of images.
Prior approaches have either ignored the problem altogether (such as in the vast majority of hand-held still cameras and in most consumer camcorders) or have involved the use of mechanical devices to dampen high frequency motions (as in most professional video cameras). Mechanical devices typically suffer from their weight, as well as their inability to measure other motion parameters, such as any form of subject motion. Image degradation may occur however, as a result of subject motion, camera motion, or any combination thereof. Digital processing of the time-varying image offers more flexibility in measuring relative motion than mechanical damping systems; and thereby offers correction or accommodation to a larger class of motion situations.
The prior art in digital processing generally approaches motion detection and measurement in a number of ways. A first is to perform a correlation of the same image region (say, one or more sensor scan lines) is made with itself but at a later time. This gives a "degree of change" but offers no insight into the cause.
A patent of interest for its teachings in this area U.S. Pat. No. 4,673,276, entitled "Blur Detecting Device for a Camera" by Yoshida et al. In that patent a correlation is performed between the output signals from a CCD image sensor at a first and second point in time. These signals are compared against threshold values to determine if allowable levels of blur have been exceeded, if so, means are provided for indicating that unacceptable blurring has occurred and that another picture should be taken.
Another patent of interest for its teachings is, U.S. Pat. No. 4,492,452 entitled "Picture BlurAlarm Device for a Camera" by Suzuki et al. there is disclosed a device that measures absolute luminance (brightness) at multiple image points, each at two disparate times. A comparison is made between the two values at each point, and a voting scheme is used wherein each difference exceeding a threshold provides an alarm signal indicating a blur condition.
Intensity-based approaches are susceptible to differences in illuminant. That is, the output values are generally not normalized for variations in intensity under different illuminant situations. Also, they generally do not yield a "direction" of motion, required if motion is to be compensated for by any means other than decreasing shutter speed.
Alternatively, "gray-level matching" is performed which finds certain predetermined patterns of luminance within the image such as typically exist at "corners", "edge-lets", or other sufficiently interesting features. These are sought in two frames, and their displacement is measured. For two different examples of such methods, see: D. T. Lawton, "Processing Translational Motion Sequences". Computer Vision, Graphics, and Image Processing, 1983, 22, 116-144, and Hans-Hellmut Nagel, "Displacement Vectors Derived from Second-order Intensity Variations in Image Sequences," Computer Vision, Graphics, and Image Processing, 1983, 21, 85-117. A sub-approach if subject motion is of interest finds the "edges" within an image and tracks their movement between frames. This is the highest level method and is the most complex in that it requires a degree of segmentation robustness and symbolic representation not required by the other methods, given their ability to work only based on intensity values. Moreover, many of these methods assume that the optical-flow field has already been calculated and is accessible; a truly prohibitive assumption here.
These aforementioned methods all suffer to various degrees from a number of problems. One is the "correspondence problem" that is the ability of finding common features within two or more images. Another is the computational burden in finding image-specific features, or alternatively in determining acceptable universally useful patch sizes and locations. Also, in general, the methods are susceptible to the difficulties of foreshortening (dimensional differences due to changes in perspective brought about by relative displacement or motion between object and viewer). Intensity-based sub-approaches are susceptible to changes in illuminant. All suffer to varying degrees by forcing a certain amount of the processing to be serial, since even though it is possible to find features in parallel, each one that is discovered must generally be processed serially since their number and nature is image-dependent.
Spatio-temporal gradient approaches measure the image intensity function's partial spatial and temporal derivatives and, based on these values, attempts to measure the "optical flow field", i.e. an array of unique motion vectors at each image site. Such approaches suffer from three principal problems: first, the determination of motion vectors at image sites is found to be mathematically ill-posed. This is because of the so-called "aperture" problem, which recognizes the difficulty in determining the velocity component perpendicular to the image gradient unless other assumptions or cues are utilized. As such, a closed form solution does not exist. The second principal problem, exacerbated by the first, is that any such computation of motion vectors at image sites presents an intractable computational burden with respect to the capabilities of present and foreseeable camera systems. The third problem lies in the difficulty of accurately measuring the spatial derivatives due to the existence of occlusion boundaries (at which the derivative fails to exist).
It would, therefore, be advantageous to have a system that could effectively measure relative motion between camera and subject, in some tenable manner, and that could compensate for the motion conditions, and/or to at least record the motion conditions such that the motion conditions could later be used as a corrective factor to a restorative phase. An initial requirement, however, is to have the measured motion simply influence the shutter speed, favoring high shutter speeds when motion is detected but otherwise allowing it to be low.